Summary of Project 3

Feature extraction, storage, running time (cost)

Submission Time

Dimensions of Extracted Features

Baseline Model

Advanced Model

Submitted File Sizes

Performance

Prediction Accuracy

Computational Efficiency

Team 9 is the only team that implemented model selection within the training process, which takes much longer time. However, to be fair in the comparison, their average training CPU time per fold per replicate is divided 7, which is the number of their candidate parameter values. They’ve also performed a 5-fold CV within each GBM training.

Fun Images

The percentages of votes to cat (1) of 18 fun images.

## NULL

Team Performance Clustering

We use Multidimensional Scaling (MDS) to cluster the performance of each team using the prediction in the first replicate.

Retrained Advanced Model

## $points
##              [,1]          [,2]
##  [1,] -0.05827984  0.3379870516
##  [2,]  0.08787685 -0.0766731754
##  [3,]  0.16314396 -0.0325527129
##  [4,] -0.17510701  0.2040679592
##  [5,]  0.10442062 -0.0406562996
##  [6,]  0.09791828 -0.0008004062
##  [7,] -0.29916904 -0.1074944152
##  [8,] -0.27641054 -0.1846644649
##  [9,]  0.10278298 -0.0552844831
## [10,]  0.09215284 -0.0195873549
## [11,]  0.16067089 -0.0243416987
## 
## $eig
##  [1] 2.996655e-01 2.141596e-01 1.470708e-01 1.189221e-01 6.357198e-02
##  [6] 4.980865e-02 2.556054e-02 1.519005e-02 1.142172e-02 9.986528e-03
## [11] 1.386641e-17
## 
## $x
## NULL
## 
## $ac
## [1] 0
## 
## $GOF
## [1] 0.5378354 0.5378354

Untuned Advanced Model

## $points
##             [,1]        [,2]
## [1,] -0.13557125  0.36873194
## [2,]  0.12319907 -0.04894373
## [3,] -0.21887867 -0.12494272
## [4,]  0.13957533 -0.05057108
## [5,]  0.02061158 -0.04208791
## [6,]  0.20248544  0.04288016
## [7,]  0.11984681 -0.02299910
## [8,] -0.25126831 -0.12206756
## 
## $eig
## [1] 2.198709e-01 1.755664e-01 4.509765e-02 3.899978e-02 2.717722e-02
## [6] 1.775751e-02 2.020690e-03 3.418264e-17
## 
## $x
## NULL
## 
## $ac
## [1] 0
## 
## $GOF
## [1] 0.7510821 0.7510821

Summary of Common Problems

• 9 teams did not have model selection
• 3 teams did not have baseline model
• 5 teams did not return the correct prediction values in either advanced model or baseline model
  • predictions are 1& 2 instead of 0&1
  • predicted probabilities/link values are returned instead of class labels.
• 4 teams did not deal with constant features, which ran into scaling error.
  • Variable(s) ‘V22’ and ‘V46’ and ‘V110’ constant. Cannot scale data.